Electronic device and object measurement method thereof

ABSTRACT

According to an embodiment, an electronic device includes a display, a memory, and a processor operatively connected to the display and the memory. The memory stores instructions that cause the processor to generate first depth information in a first direction from an object, generate a first point cloud of the object based on the first depth information, generate a first bounding box containing one or more points of the first point cloud, generate second depth information in a second direction from the object, the second direction being different from the first direction, generate a second point cloud of the object based on the second depth information, generate a second bounding box containing one or more points of the second point cloud, generate a third bounding box by combining the first and second bounding boxes, and display the third bounding box on the display. Certain other embodiments are also possible.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is based on and claims priority under 35 U.S.C. 119 toKorean Patent Application No. 10-2019-0019536 filed on Feb. 19, 2019, inthe Korean Intellectual Property Office, the disclosures of which areherein incorporated by reference in their entireties.

TECHNICAL FIELD

The instant disclosure generally relates to an electronic device and, inparticular, to an electronic device that is capable of measuring thesize of an object using information of neighboring objects.

BACKGROUND

Using augmented reality technology, a 3-dimensional virtual image may besuperimposed on a real-world background image captured by a camera, andthe combined image may be provided to the user. A portable electronicdevice (hereinafter, referred to as “electronic device”) such as asmartphone can be designed to provide users with various types ofcontent based on augmented reality technology.

An exemplary application of the augmented reality technology is toprovide information on a real-world object using virtual images. Onesuch type of information on the real-world object is the size of theobject. For example, the user may acquire size information on a box forpacking a cup using the electronic device that is capable of taking animage of the cup, measuring the size of the cup from the image, anddisplaying the size information of the box.

SUMMARY

Conventionally, object size measurement may be achieved by measuring adistance between points designated by the user on a captured real-worldimage. For example, when the electronic device displays a preview imagefrom its camera, the user may sequentially select the top and bottom endpoints of an object to measure the distance between the two selectedpoints. The size of the object may then be determined based on theresult of the measurement.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments.

According to a disclosed embodiment, an electronic device includes adisplay, a memory, and a processor operatively connected to the displayand the memory. The memory stores instructions that, when executed bythe processor, cause the processor to generate first depth informationin a first direction from an object, generate a first point cloud of theobject based on the first depth information, generate a first boundingbox containing one or more points of the first point cloud, generatesecond depth information in a second direction from the object, thesecond direction being different from the first direction, generate asecond point cloud of the object based on the second depth information,generate a second bounding box containing one or more points of thesecond point cloud, generate a third bounding box by combining the firstand second bounding boxes, and display the third bounding box on thedisplay.

According to a disclosed embodiment, an object measurement method of anelectronic device includes generating first depth information in a firstdirection from an object, generating a first point cloud of the objectbased on the first depth information, generating a first bounding boxcontaining one or more points of the first point cloud, generatingsecond depth information in a second direction from the object, thesecond direction being different from the first direction, generating asecond point cloud of the object based on the second depth information,generating a second bounding box containing one or more points of thesecond point cloud, generating a third bounding box by combining thefirst and second bounding boxes, and displaying the third bounding box.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, the same or similar reference numerals may be used forthe same or similar components.

The above and other aspects, features, and advantages of certainembodiments of the present disclosure will be more apparent from thefollowing description taken in conjunction with the accompanyingdrawings, in which:

FIG. 1 is a block diagram illustrating a configuration of an electronicdevice in a network environment according to a disclosed embodiment;

FIG. 2 is a diagram illustrating an electronic device and an objectaccording to a disclosed embodiment;

FIG. 3 is a block diagram illustrating an electronic device according toa disclosed embodiment;

FIG. 4 is a block diagram illustrating hardware and software of anelectronic device for generating 3-dimensional coordinates of an objectaccording to a disclosed embodiment;

FIG. 5 is a diagram illustrating an exemplary screen display forexplaining a procedure for an electronic device to guide targeting of anobject to be measured according to a disclosed embodiment;

FIG. 6 is a diagram illustrating an exemplary screen display forexplaining a procedure for an electronic device to generate a pointcloud of an object according to a disclosed embodiment;

FIGS. 7A and 7B are diagrams illustrating exemplary screen displays forexplaining a procedure for an electronic device to remove a referencesurface from a point cloud according to a disclosed embodiment;

FIG. 8 is a diagram illustrating an exemplary screen display forexplaining a procedure for an electronic device to generate a3-dimensional bounding box according to a disclosed embodiment;

FIGS. 9A to 9D are diagrams illustrating exemplary screen displays forexplaining a procedure for an electronic device to display objectmeasurement-related guide information according to a disclosedembodiment;

FIG. 10 is a diagram illustrating an exemplary screen display forexplaining a point cloud of an object that is generated by an electronicdevice according to a disclosed embodiment;

FIG. 11 is a diagram illustrating an exemplary screen display forexplaining a procedure for an electronic device to generate two boundingboxes for an object according to a disclosed embodiment;

FIG. 12 is a diagram illustrating an exemplary screen display forexplaining a procedure for an electronic device to update a bounding boxaccording to a disclosed embodiment;

FIGS. 13A and 13B are diagrams illustrating exemplary screen displaysfor explaining a procedure for an electronic device to generate abounding box of a 2-dimensional object according to a disclosedembodiment; and

FIG. 14 is a flowchart illustrating an object measurement method of anelectronic device according to a disclosed embodiment.

DETAILED DESCRIPTION

Certain disclosed embodiments aim to provide an electronic device andobject measurement method thereof that is capable of automatically andaccurately measuring the size of a real-world object.

FIG. 1 is a block diagram illustrating an electronic device 101 in anetwork environment 100 according to various embodiments. Referring toFIG. 1, the electronic device 101 in the network environment 100 maycommunicate with an electronic device 102 via a first network 198 (e.g.,a short-range wireless communication network), or an electronic device104 or a server 108 via a second network 199 (e.g., a long-rangewireless communication network). According to an embodiment, theelectronic device 101 may communicate with the electronic device 104 viathe server 108. According to an embodiment, the electronic device 101may include a processor 120, memory 130, an input device 150, a soundoutput device 155, a display device 160, an audio module 170, a sensormodule 176, an interface 177, a haptic module 179, a camera module 180,a power management module 188, a battery 189, a communication module190, a subscriber identification module (SIM) 196, or an antenna module197. In some embodiments, at least one (e.g., the display device 160 orthe camera module 180) of the components may be omitted from theelectronic device 101, or one or more other components may be added inthe electronic device 101. In some embodiments, some of the componentsmay be implemented as single integrated circuitry. For example, thesensor module 176 (e.g., a fingerprint sensor, an iris sensor, or anilluminance sensor) may be implemented as embedded in the display device160 (e.g., a display).

The processor 120 may execute, for example, software (e.g., a program140) to control at least one other component (e.g., a hardware orsoftware component) of the electronic device 101 coupled with theprocessor 120, and may perform various data processing or computation.According to one embodiment, as at least part of the data processing orcomputation, the processor 120 may load a command or data received fromanother component (e.g., the sensor module 176 or the communicationmodule 190) in volatile memory 132, process the command or the datastored in the volatile memory 132, and store resulting data innon-volatile memory 134. According to an embodiment, the processor 120may include a main processor 121 (e.g., a central processing unit (CPU)or an application processor (AP)), and an auxiliary processor 123 (e.g.,a graphics processing unit (GPU), an image signal processor (ISP), asensor hub processor, or a communication processor (CP)) that isoperable independently from, or in conjunction with, the main processor121. Additionally or alternatively, the auxiliary processor 123 may beadapted to consume less power than the main processor 121, or to bespecific to a specified function. The auxiliary processor 123 may beimplemented as separate from, or as part of the main processor 121.

The auxiliary processor 123 may control at least some of functions orstates related to at least one component (e.g., the display device 160,the sensor module 176, or the communication module 190) among thecomponents of the electronic device 101, instead of the main processor121 while the main processor 121 is in an inactive (e.g., sleep) state,or together with the main processor 121 while the main processor 121 isin an active state (e.g., executing an application). According to anembodiment, the auxiliary processor 123 (e.g., an image signal processoror a communication processor) may be implemented as part of anothercomponent (e.g., the camera module 180 or the communication module 190)functionally related to the auxiliary processor 123.

The memory 130 may store various data used by at least one component(e.g., the processor 120 or the sensor module 176) of the electronicdevice 101. The various data may include, for example, software (e.g.,the program 140) and input data or output data for a command relatedthereto. The memory 130 may include the volatile memory 132 or thenon-volatile memory 134.

The program 140 may be stored in the memory 130 as software, and mayinclude, for example, an operating system (OS) 142, middleware 144, oran application 146.

The input device 150 may receive a command or data to be used by othercomponent (e.g., the processor 120) of the electronic device 101, fromthe outside (e.g., a user) of the electronic device 101. The inputdevice 150 may include, for example, a microphone, a mouse, or akeyboard.

The sound output device 155 may output sound signals to the outside ofthe electronic device 101. The sound output device 155 may include, forexample, a speaker or a receiver. The speaker may be used for generalpurposes, such as playing multimedia or playing record, and the receivermay be used for an incoming calls. According to an embodiment, thereceiver may be implemented as separate from, or as part of the speaker.

The display device 160 may visually provide information to the outside(e.g., a user) of the electronic device 101. The display device 160 mayinclude, for example, a display, a hologram device, or a projector andcontrol circuitry to control a corresponding one of the display,hologram device, and projector. According to an embodiment, the displaydevice 160 may include touch circuitry adapted to detect a touch, orsensor circuitry (e.g., a pressure sensor) adapted to measure theintensity of force incurred by the touch.

The audio module 170 may convert a sound into an electrical signal andvice versa. According to an embodiment, the audio module 170 may obtainthe sound via the input device 150, or output the sound via the soundoutput device 155 or a headphone of an external electronic device (e.g.,an electronic device 102) directly (e.g., wiredly) or wirelessly coupledwith the electronic device 101.

The sensor module 176 may detect an operational state (e.g., power ortemperature) of the electronic device 101 or an environmental state(e.g., a state of a user) external to the electronic device 101, andthen generate an electrical signal or data value corresponding to thedetected state. According to an embodiment, the sensor module 176 mayinclude, for example, a gesture sensor, a gyro sensor, an atmosphericpressure sensor, a magnetic sensor, an acceleration sensor, a gripsensor, a proximity sensor, a color sensor, an infrared (IR) sensor, abiometric sensor, a temperature sensor, a humidity sensor, or anilluminance sensor.

The interface 177 may support one or more specified protocols to be usedfor the electronic device 101 to be coupled with the external electronicdevice (e.g., the electronic device 102) directly (e.g., wiredly) orwirelessly. According to an embodiment, the interface 177 may include,for example, a high definition multimedia interface (HDMI), a universalserial bus (USB) interface, a secure digital (SD) card interface, or anaudio interface.

A connecting terminal 178 may include a connector via which theelectronic device 101 may be physically connected with the externalelectronic device (e.g., the electronic device 102). According to anembodiment, the connecting terminal 178 may include, for example, a HDMIconnector, a USB connector, a SD card connector, or an audio connector(e.g., a headphone connector),

The haptic module 179 may convert an electrical signal into a mechanicalstimulus (e.g., a vibration or a movement) or electrical stimulus whichmay be recognized by a user via his tactile sensation or kinestheticsensation. According to an embodiment, the haptic module 179 mayinclude, for example, a motor, a piezoelectric element, or an electricstimulator.

The camera module 180 may capture a still image or moving images.According to an embodiment, the camera module 180 may include one ormore lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to theelectronic device 101. According to one embodiment, the power managementmodule 188 may be implemented as at least part of, for example, a powermanagement integrated circuit (PMIC).

The battery 189 may supply power to at least one component of theelectronic device 101. According to an embodiment, the battery 189 mayinclude, for example, a primary cell which is not rechargeable, asecondary cell which is rechargeable, or a fuel cell.

The communication module 190 may support establishing a direct (e.g.,wired) communication channel or a wireless communication channel betweenthe electronic device 101 and the external electronic device (e.g., theelectronic device 102, the electronic device 104, or the server 108) andperforming communication via the established communication channel. Thecommunication module 190 may include one or more communicationprocessors that are operable independently from the processor 120 (e.g.,the application processor (AP)) and supports a direct (e.g., wired)communication or a wireless communication. According to an embodiment,the communication module 190 may include a wireless communication module192 (e.g., a cellular communication module, a short-range wirelesscommunication module, or a global navigation satellite system (GNSS)communication module) or a wired communication module 194 (e.g., a localarea network (LAN) communication module or a power line communication(PLC) module). A corresponding one of these communication modules maycommunicate with the external electronic device via the first network198 (e.g., a short-range communication network, such as Bluetooth™,wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA))or the second network 199 (e.g., a long-range communication network,such as a cellular network, the Internet, or a computer network (e.g.,LAN or wide area network (WAN)). These various types of communicationmodules may be implemented as a single component (e.g., a single chip),or may be implemented as multi components (e.g., multi chips) separatefrom each other. The wireless communication module 192 may identify andauthenticate the electronic device 101 in a communication network, suchas the first network 198 or the second network 199, using subscriberinformation (e.g., international mobile subscriber identity (IMSI))stored in the subscriber identification module 196.

The antenna module 197 may transmit or receive a signal or power to orfrom the outside (e.g., the external electronic device) of theelectronic device 101. According to an embodiment, the antenna module197 may include one or more antennas, and, therefrom, at least oneantenna appropriate for a communication scheme used in the communicationnetwork, such as the first network 198 or the second network 199, may beselected, for example, by the communication module 190 (e.g., thewireless communication module 192). The signal or the power may then betransmitted or received between the communication module 190 and theexternal electronic device via the selected at least one antenna.

At least some of the above-described components may be coupled mutuallyand communicate signals (e.g., commands or data) therebetween via aninter-peripheral communication scheme (e.g., a bus, general purposeinput and output (GPIO), serial peripheral interface (SPI), or mobileindustry processor interface (MIPI)).

According to an embodiment, commands or data may be transmitted orreceived between the electronic device 101 and the external electronicdevice 104 via the server 108 coupled with the second network 199. Eachof the electronic devices 102 and 104 may be a device of a same type as,or a different type, from the electronic device 101. According to anembodiment, all or some of operations to be executed at the electronicdevice 101 may be executed at one or more of the external electronicdevices 102, 104, or 108. For example, if the electronic device 101should perform a function or a service automatically, or in response toa request from a user or another device, the electronic device 101,instead of, or in addition to, executing the function or the service,may request the one or more external electronic devices to perform atleast part of the function or the service. The one or more externalelectronic devices receiving the request may perform the at least partof the function or the service requested, or an additional function oran additional service related to the request, and transfer an outcome ofthe performing to the electronic device 101. The electronic device 101may provide the outcome, with or without further processing of theoutcome, as at least part of a reply to the request. To that end, acloud computing, distributed computing, or client-server computingtechnology may be used, for example.

The electronic device according to various embodiments may be one ofvarious types of electronic devices. The electronic devices may include,for example, a portable communication device (e.g., a smart phone), acomputer device, a portable multimedia device, a portable medicaldevice, a camera, a wearable device, or a home appliance. According toan embodiment of the disclosure, the electronic devices are not limitedto those described above.

It should be appreciated that various embodiments of the presentdisclosure and the terms used therein are not intended to limit thetechnological features set forth herein to particular embodiments andinclude various changes, equivalents, or replacements for acorresponding embodiment. With regard to the description of thedrawings, similar reference numerals may be used to refer to similar orrelated elements. It is to be understood that a singular form of a nouncorresponding to an item may include one or more of the things, unlessthe relevant context clearly indicates otherwise. As used herein, eachof such phrases as “A or B,” “at least one of A and B,” “at least one ofA or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least oneof A, B, or C,” may include all possible combinations of the itemsenumerated together in a corresponding one of the phrases. As usedherein, such terms as “1st” and “2nd,” or “first” and “second” may beused to simply distinguish a corresponding component from another, anddoes not limit the components in other aspect (e.g., importance ororder). It is to be understood that if an element (e.g., a firstelement) is referred to, with or without the term “operatively” or“communicatively”, as “coupled with,” “coupled to,” “connected with,” or“connected to” another element (e.g., a second element), it means thatthe element may be coupled with the other element directly (e.g.,wiredly), wirelessly, or via a third element.

As used herein, the term “module” may include a unit implemented inhardware, software, or firmware, and may interchangeably be used withother terms, for example, “logic,” “logic block,” “part,” or“circuitry”. A module may be a single integral component, or a minimumunit or part thereof, adapted to perform one or more functions. Forexample, according to an embodiment, the module may be implemented in aform of an application-specific integrated circuit (ASIC).

Various embodiments as set forth herein may be implemented as software(e.g., the program 140) including one or more instructions that arestored in a storage medium (e.g., internal memory 136 or external memory138) that is readable by a machine (e.g., the electronic device 101).For example, a processor (e.g., the processor 120) of the machine (e.g.,the electronic device 101) may invoke at least one of the one or moreinstructions stored in the storage medium, and execute it, with orwithout using one or more other components under the control of theprocessor. This allows the machine to be operated to perform at leastone function according to the at least one instruction invoked. The oneor more instructions may include a code generated by a compiler or acode executable by an interpreter. The machine-readable storage mediummay be provided in the form of a non-transitory storage medium. Wherein,the term “non-transitory” simply means that the storage medium is atangible device, and does not include a signal (e.g., an electromagneticwave), but this term does not differentiate between where data issemi-permanently stored in the storage medium and where the data istemporarily stored in the storage medium.

According to an embodiment, a method according to various embodiments ofthe disclosure may be included and provided in a computer programproduct. The computer program product may be traded as a product betweena seller and a buyer. The computer program product may be distributed inthe form of a machine-readable storage medium (e.g., compact disc readonly memory (CD-ROM)), or be distributed (e.g., downloaded or uploaded)online via an application store (e.g., Play Store™), or between two userdevices (e.g., smart phones) directly. If distributed online, at leastpart of the computer program product may be temporarily generated or atleast temporarily stored in the machine-readable storage medium, such asmemory of the manufacturer's server, a server of the application store,or a relay server.

According to various embodiments, each component (e.g., a module or aprogram) of the above-described components may include a single entityor multiple entities. According to various embodiments, one or more ofthe above-described components may be omitted, or one or more othercomponents may be added. Alternatively or additionally, a plurality ofcomponents (e.g., modules or programs) may be integrated into a singlecomponent. In such a case, according to various embodiments, theintegrated component may still perform one or more functions of each ofthe plurality of components in the same or similar manner as they areperformed by a corresponding one of the plurality of components beforethe integration. According to various embodiments, operations performedby the module, the program, or another component may be carried outsequentially, in parallel, repeatedly, or heuristically, or one or moreof the operations may be executed in a different order or omitted, orone or more other operations may be added.

FIG. 2 is a diagram illustrating an electronic device and an objectaccording to a disclosed embodiment.

According to an embodiment, the electronic device 210 may be, but it isnot limited to, a portable electronic device such as a smartphone and atablet personal computer (PC). The electronic device 210 may include atleast one sensor (e.g., time of flight (ToF) sensor 211 and red greenblue (RGB) image sensor 212) for taking images within a predeterminedview angle and creating depth information (or depth map) of an object220 included in the taken image.

A method for the electronic device 210 to generate the depth informationof the object 220 using the ToF sensor 211 and/or the RGB image sensor212 is described in detail with reference to FIGS. 3 to 14.

The object 220 of which the depth information can be obtained is notlimited by the embodiment shown in FIG. 2. Although FIG. 2 is directedto an example of taking an image of a cup 221 placed on a table 222, theelectronic device 210 may also take images of other objects such ashuman bodies to obtain the depth information of the objects.

Although the description in the instant disclosure is directed to theexemplary case of obtaining the depth information of the cup 221 in thetaken image and automatically displaying a bounding box and size of thecup 221 that is automatically measured based on the depth information(e.g., FIGS. 5 to 12), the methods of the certain disclosed embodimentsmay be used for measuring sizes of various types of 3-dimensional or2-dimensional objects (e.g., pictures drawn on flat surfaces).

The electronic device 210 may create a point cloud based on the depthinformation of the object 220 in the taken image and proceed to create abounding box of the point cloud.

According to an embodiment, the electronic device 210 may createmultiple point clouds (e.g., first point cloud and second point cloud)based on the depth information (e.g., first depth information and seconddepth information) obtained at two or more locations (e.g., firstdirection and second direction), create multiple bounding boxes (e.g.,first to third bounding boxes) from the point clouds, and display thebounding boxes.

After displaying the created bounding boxes, if it is determined that abounding box is inaccurate, the electronic device 210 may display guideinformation prompting the user to move the electronic device 210 to takean additional image (or depth information) of the corresponding object220 at a different location.

FIG. 3 is a block diagram illustrating an electronic device according toa disclosed embodiment.

In reference to FIG. 3, the electronic device 300 may include a ToFsensor 310, an RGB sensor 320, a display 330, a processor 340, and amemory 350 of which at least one may be omitted or replaced. Theelectronic device 300 may include at least some of the configurationand/or functionality of the electronic device of FIG. 1.

The components constituting the electronic device 300 are contained in ahousing, and some components that emit (e.g., display 330) light orreceive (e.g., ToF sensor 310) light from the exterior of the housingmay be partially exposed to the outside of the housing.

According to an embodiment, the display 330 may display images and maybe implemented with at least one of various types of display devicessuch as liquid crystal display (LCD) or light-emitting diode (LED)display. The display 330 may include at least part of the configurationand/or functionality of the display device 160 of FIG. 1. The display330 may be exposed through an opening formed in one surface (e.g., frontsurface) of the housing.

The display 330 may be a touchscreen display that is capable of sensingtouches or hovering (i.e. proximity touch) gestures made by, forexample, one or more fingers of the user or by a stylus pen.

According to an embodiment, the ToF sensor 310 is capable of obtainingdepth information of an object using ToF technology. For example, theToF sensor 310 may emit infrared rays and calculate the times of arrivalof the rays reflected from the object to measure the distances from theToF sensor 310 to the parts of the object reflecting the light. Based onthe distances measured by the ToF sensor 310, it is possible generatedepth information of the object.

The ToF sensor 310 may generate depth information from multiple imageframes acquired from multiple locations in multiple directions using ToFtechnology.

According to an embodiment, the RGB image sensor 320 may take images ofan object and generate image data including red (R), green (G), and blue(blue) data. The RGB image sensor 320 may acquire the image data usingone of various technologies such as charge coupled device (CCD) andcomplementary metal oxide semiconductor (CMOS).

According to an embodiment, the processor 340 may generate depthinformation of the object based on two pieces of image data of theobject that are acquired by the RGB image sensor in differentdirections.

According to an embodiment, the electronic device 300 may include atleast one of the ToF sensor 310 and the RGB image sensor 320 and/or mayinclude an additional sensor that is capable of generating depthinformation of the object in addition to the ToF sensor 310 and the RGBsensor 320.

According to an embodiment, the memory may include, but it is notlimited to, volatile and non-volatile memories. The memory 350 mayinclude at least some of the configuration and/or functionality of thememory of FIG. 1. The memory 350 may also store at least some ofprograms of FIG. 1.

The memory 350 may store various instructions executable by theprocessor 340. The instructions may include arithmetic and logicoperations, instructions for manipulating data, and input/output controlcommands.

According to an embodiment, the processor 340 may be configured tocontrol the components of the electronic device 300 to performcommunication-related operations and data processing operations and mayinclude at least some of the configurations and/or functionality of theprocessor 120 of FIG. 1. The processor 340 may be functionally,operatively, and/or electrically connected to the components of theelectronic device 300, i.e., the ToF sensor 310, the RGB image sensor320, the display 330, and the memory 350. The processor 340 may includea microprocessor or any suitable type of processing circuitry, such asone or more general-purpose processors (e.g., ARM-based processors), aDigital Signal Processor (DSP), a Programmable Logic Device (PLD), anApplication-Specific Integrated Circuit (ASIC), a Field-ProgrammableGate Array (FPGA), a Graphical Processing Unit (GPU), a video cardcontroller, etc. In addition, it would be recognized that when a generalpurpose computer accesses code for implementing the processing shownherein, the execution of the code transforms the general purposecomputer into a special purpose computer for executing the processingshown herein. Certain of the functions and steps provided in the Figuresmay be implemented in hardware, software or a combination of both andmay be performed in whole or in part within the programmed instructionsof a computer. No claim element herein is to be construed under theprovisions of 35 U.S.C. § 112(f), unless the element is expresslyrecited using the phrase “means for.” In addition, an artisanunderstands and appreciates that a “processor” or “microprocessor” maybe hardware in the claimed disclosure. Under the broadest reasonableinterpretation, the appended claims are statutory subject matter incompliance with 35 U.S.C. § 101.

The processor 340 are not limited to perform arithmetic operations anddata processing functions of the electronic device 300. Certaindisclosed embodiments are directed to a method for automatically andaccurately measuring the size of a real-world object and guide the userto a more accurate measurement method. The operations of the processor340 to be described below may be performed after correspondinginstructions stored in the memory 350 are loaded.

According to an embodiment, the processor 340 may generate first depthinformation in a first direction from the object.

According to an embodiment, the processor 340 may generate the firstdepth information based on a first ToF frame taken by the ToF sensor 310located in the first direction from the object. The ToF sensor 310 maymeasure distances between points of each object located within a viewangle, and the processor 340 may generate the first depth informationbased on the distances.

According to an alternative embodiment, the processor 340 may generatethe first depth information based on the difference between a third RGBframe that the RGB image sensor 320 acquires in a third direction fromthe object and a first RGB frame that the RGB image sensor 320 acquiresin the first direction from the object. The first and third directionsfrom the object may form a predetermined angle, and thus the first andthird RGB frames may be image frames taken in different directions fromthe object. Accordingly, the processor 340 may acquire the first depthinformation of the object based on the difference between the first andthird RGB frames acquired in different directions from the object.

According to an embodiment, the processor 340 may generate a first pointcloud of the object based on the first depth information. A point cloudinclude points captured on the surface of the 3-dimensional object. Theindividual points constituting the point cloud may be identified bycoordinate information relative to a reference point.

According to an embodiment, the processor 340 may generate a first pointcloud with only the points above a reference surface. That is, in thegenerated point cloud, the processor 340 may remove the points on thereference surface.

In the case of measuring an object placed on a specific referencesurface (e.g., table and ground), because the reference surface isincluded in the image frames (e.g., ToF frame and RGB image frame), thedepth information and point cloud of the reference surface may beincluded. The processor 340 may remove the point cloud of the referencesurface to generate a bounding box for only the object.

According to an embodiment, the processor 340 may detect points havingthe same normal vector (or being within an error range) among the pointsconstituting the point cloud. The processor 340 may then identify thepoints adjacent to each other, among the points having the same normalvector, based on their coordinate information. According to analternative embodiment, the processor 340 may determine a surface havingthe largest number of points with the same coordinate value in the zaxis (e.g., z axis in FIG. 2) as the reference surface in the generatepoint cloud.

By removing the reference surface detected as above, it is possible togenerate the first point cloud composed of the points corresponding tothe object. In the case where multiple objects exist in a taken imageframe (e.g., ToF frame and RGB image frame), the objects may beseparated by removing the reference surface. The processor 340 maygenerate point clouds and bounding boxes for the separated objects.

According to an embodiment, the processor 340 may generate a firstbounding box based on the first point cloud. A bounding box is formed asthe smallest cuboid that the object can fit in. For example, theprocessor 340 may generate the first bounding box based on the smallestand largest coordinate values in the x, y, and z axes that are obtainedfrom the coordinate information of the individual points of the firstpoint cloud.

According to an embodiment, in the case where multiple point clouds aregenerated for multiple objects included in the taken image frame (e.g.,ToF frame and RGB image frame), due to the removal of the referencesurface, the processor 340 may generate a bounding box for each object.

According to an embodiment, the processor may display the first boundingbox on the display 330. Here, the processor 340 may display the firstbounding box such that the first bounding box is overlaid on the imageof the object that is taken by the RGB image sensor 320. In the casewhere there are multiple objects, bounding boxes for each object areoverlaid on the respective objects.

The first bounding box generated as above may be inaccurate because itis generated based on only one set of depth information. The accuracy islow because it is difficult to measure 3-D depth information from onlythe first direction. According to a disclosed embodiment, the electronicdevice 300 may calculate a bounding box and numerical values moreaccurately using multiple sets of depth information acquired indifferent directions.

According to an embodiment, the processor 340 may generate second depthinformation in a second direction from the object. Here, the seconddirection differs from the first direction and may form a predeterminedangle with the first direction horizontally or vertically with respectto the object.

The second depth information may be generated in the same manner as thatused for the first depth information. For example, the processor 340 maygenerate the second depth information based on a second ToF frameacquired by the ToF sensor 310 in the second direction from the object.The processor 340 may also generate the second depth information basedon the difference between the first RGB frame that the RGB sensor 320acquires in the first direction from the object and a second RGB framethat the RGB image sensor 320 acquires in the second direction from theobject.

According to an embodiment, the processor 340 may generate a secondpoint cloud of the object based on the second depth information. Theprocessor 340 may generate the second point cloud with only the pointsabove a reference surface among the point cloud generated based on thesecond depth information.

According to an embodiment, the processor 340 may generate a secondbounding box based on the second point cloud. The second bounding boxmay differ in part from the first bounding box. For example, the secondbounding box may have a dimension extended (or shrunken) in the x, y,and z directions with respect to the first bounding box. Here, theprocessor 340 may display the second bounding box such that the secondbounding box is overlaid on the image of the object that is taken by theRGB image sensor 320.

According to an embodiment, a third bounding box may be generated bycombining the first and second bounding boxes. As described above, thefirst and second bounding boxes differ in size because they are measuredin different directions from the object.

The third bounding box may have a shape of a cuboid that can fit thefirst and second bounding boxes. That is, the third bounding box may begenerated based on the smallest and largest of the x-, y-, and z-axiscoordinate values of the first and second bounding boxes.

According to an embodiment, the processor 340 may display the thirdbounding box on the display 330. The processor 340 may display the thirdbounding box such that the third bounding box is overlaid on the imageof the object that is acquired by the RGB image sensor 320.

According to an embodiment, the processor 340 may display the thirdbounding box along with numeric information. The numeric information mayinclude edge lengths and the volume of the third bounding box. Theprocessor 340 may generate the numeric information based on the smallestand largest coordinate values of the point clouds contained in the thirdbounding box.

According to an embodiment, if it is determined that the first boundingbox is inaccurate, the processor 340 may display guide information,which prompts for generation of another bounding box in a differentdirection, on the display 330.

According to an embodiment, if the first bounding box differs in partfrom a region where the actual object is included, the processor 340 mayidentify the first bounding box as inaccurate and display guideinformation prompting generation of the second bounding box in adifferent direction.

According to an alternative embodiment, if it is determined that theobject extracted from the first RGB image is partly out of the viewingangle of the RGB image sensor 320, the processor 340 may display guideinformation prompting a shift in the location of the electronic device300.

According to an alternative embodiment, if the number of the pointsconstituting the first point clouds is less than a threshold, theprocessor 340 may display guide information prompting a shift in thelocation of the electronic device 300 closer to the object.

According to an alternative embodiment, if the number of first pointclouds acquired from the first ToF frame taken by the ToF sensor 310 isless than a threshold, the processor 340 may display guide informationprompting acquisition of depth information with the RGB image sensor320.

How to display the guide information is described later with referenceto FIGS. 9A to 9D.

According to an embodiment, the electronic device 300 may generate abounding box for a 2-dimensional object (e.g., rectangle and ellipse) ona plane.

If it fails to detect any 3-dimensional objects in the image framescaptured by the ToF sensor 310 and/or the RGB image sensor 320 or if theuser selects a 2-dimensional object measurement option, the processor340 may perform a process to acquire depth information of a2-dimensional object.

The processor 340 may generate a point cloud based on the depthinformation of the 2-dimensional object and proceed to generate a pointcloud mask by cropping the points forming the shape of the 2-dimensionalobject among the generated point cloud.

The processor 340 may calculate the smallest and largest x- and y-axiscoordinate values of the points of the point cloud mask and generate abounding box in the form of a 2-dimensional shape (e.g., rectangle)based on the calculated coordinate values.

The processor 340 may display the generated bounding box on the display330 such that the bounding box is overlaid on the image of the objectalong with numeric information including the length of at least one sideof the bounding box and/or the area of the bounding box.

FIG. 4 is a block diagram illustrating hardware and software of anelectronic device for generating 3-dimensional coordinates of an objectaccording to a disclosed embodiment.

In reference to FIG. 4, the electronic device 400 may include thefollowing hardware components: a processor 440, an RGB image sensor 420,a ToF sensor 410, an inertial measurement unit (IMU) sensor 460, adisplay 430, and a memory 450. The processor 440 may execute one or moreprocesses or threads for generating a point cloud and/or bounding box ofan object.

According to an embodiment, if the user opts to execute an objectmeasurement function, the electronic device 400 may display an image ofan object that is taken by the RGB image sensor 420, measurementinformation (e.g., point cloud, bounding box, and numeric information),and guide information through a graphic user interface 441.

The ToF sensor 410 may generate and send at least one ToF frame of theobject to the processor 440, and the processor 440 may include a depthmap generator 442 to generate depth information based on the ToF frame.The electronic device 400 may include additional sensors capable ofgenerating the depth information.

The processor 440 may include a camera pose re-localizer 443 tore-localize a camera pose of the RGB image sensor 420.

The IMU sensor 460 may acquire various pieces of information related tothe location and movement of the electronic device 400 such as rotationamount, acceleration, and location information of the electronic device400 using at least one of a gyroscope, an accelerometer, and anelectromagnetic sensor. The processor 440 may include an IMU noisefilter 444 that is capable of filtering noise from information acquiredby the IMU sensor 460.

The processor 440 may re-localize the camera pose by means of the camerapose re-localizer 443 using the sensing value of the IMU sensor 460. Atthis time, the processor 440 may reorganize image frames (e.g., ToFframe).

The processor 440 may include a local iterative closest point (ICP)module 446 that is capable of generating a point cloud based on thedepth information generated by the depth map generator 442 and thecamera pose re-localized by the camera pose re-localizer 443.

The local ICP module 446 may generate a bounding box of the object basedon the point cloud.

The processor 440 may display the bounding box and numeric information(e.g., length, area, and volume) of the bounding box on the display 430with a graphic user interface and may store the bounding box and numericinformation in the memory 450.

FIG. 5 is a diagram illustrating an exemplary screen display forexplaining a procedure for an electronic device to guide targeting of anobject to be measured according to a disclosed embodiment.

In reference to FIG. 5, the user may operate the electronic device 500to display a real time image of the object, which is taken by the RGBimage sensor (e.g., RGB image sensor 320 in FIG. 3), on the display 530.The processor (e.g., processor 340 in FIG. 3) may display a targetingimage 565 for guiding arrangement of the object to be measured to be atthe center of the screen.

If the user adjusts the angle of the electronic device 500 such that thetargeting image 565 is placed at the center of the object, the processormay acquire a first ToF frame by means of a ToF sensor (e.g., ToF sensor310 in FIG. 3) automatically or in response to a touch input made by theuser. The first ToF frame may include information on a cup 571 as theobject to be measured and a table 572 on which the cup 571 is placed.

The processor may also acquire an RGB image frame by means of an RGBimage sensor.

The processor may acquire first depth information from the first ToFframe. The first depth information may include information on the depthsof all objects (e.g., cup 571 and table 572). In this case, the objectsmay be recognized as a single object because they are in contact witheach other.

FIG. 6 is a diagram illustrating an exemplary screen display forexplaining a procedure for an electronic device to generate a pointcloud of an object according to a disclosed embodiment.

According to an embodiment, the processor (e.g., processor 340 in FIG.5) may generate a first point cloud 680 based on first depth informationand display the first point cloud 680 on the display 630.

The point cloud may include points constituting a surface of the3-dimensional object. The points constituting the point cloud may eachhave coordinate information relative to a reference point.

In reference to FIG. 6, the processor may display the points of thefirst point cloud 680 on an image taken by the RGB image sensor (e.g.,RGB image sensor 320 in FIG. 3) such that the points are overlaid on theobjects in the image. Here, the first point cloud 680 may include thepoints measured on the cup 671 and the table 672 as the objects in thescreen.

FIGS. 7A and 7B are diagrams illustrating exemplary screen displays forexplaining a procedure for an electronic device to remove a referencesurface from a point cloud according to a disclosed embodiment.

In reference to FIG. 7A, a first point cloud that is provisionallyacquired in the process of measuring the size of the cup 771 placed on areference surface such as a table may include information on both thecup 771 and the table, even though the object that the user intends tomeasure is the cup 771.

According to an embodiment, the processor (e.g., processor 340 in FIG.3) may generate the first point cloud 781 including only the pointsabove the reference surface among the provisional first point cloudgenerated based on the first depth information. That is, the processormay remove the points at or below the reference surface (e.g., table)among the provisionally generated first point cloud.

According to an embodiment, the processor may detect points having thesame normal vector (or being within an error range) in the pointsconstituting the first point cloud 781 and identify the points adjacentto each other, among the points having the same normal vector, based ontheir coordinate information. Because the points corresponding to thetable are all arranged on the same plane, their normal vectors may beperpendicular to the z axis. The processor may identify the pointsadjacent to each other and having the same normal vector (or beingwithin an error range) as the reference surface.

According to an alternative embodiment, the processor may determine asurface having the largest number of points with the same z axiscoordinate value (or within an error range) in the provisionallygenerated point cloud as the reference surface.

In reference to FIG. 7A, the point cloud 781 of the cup 771 is obtainedby removing the reference surface from the provisional point cloud shownin FIG. 6.

In reference to FIG. 7B, in the case where multiple objects are includedin the captured image frame (e.g., ToF frame and RGB image frame), thepoint clouds 781 a and 781 b of respective objects 771 a and 771 b maybe separated from each other by removing the reference surface.

In this case, the processor may generate the first point clouds 781 aand 781 b of the separated objects 771 a and 771 b and then proceed togenerate and display bounding boxes for each object.

FIG. 8 is a diagram illustrating an exemplary screen display forexplaining a procedure for the electronic device to generate a3-dimensional bounding box according to a disclosed embodiment.

According to an embodiment, a processor (e.g., processor 340 in FIG. 3)may generate a first bounding box 890 based on a first point cloud(e.g., first point cloud 781 in FIG. 7A). The bounding box may begenerated in the shape of the smallest cuboid that the object can fitin. For example, the processor may generate the first bounding box 890based on the smallest and largest coordinate values in the x, y, and zaxes that are obtained from the coordinate information of the individualpoints of the first point cloud.

The processor may display the first bounding box 890 to be overlaid onthe image of the object 871.

The processor may display the first bounding box 890 along with numericinformation 899. Here, the numeric information 899 may include edgelengths and the volume of the first bounding box 890. The processor maygenerate the numeric information 899 based on the smallest and largestcoordinate values of the point cloud contained in the first bounding box890.

In reference to FIG. 8, the bounding box generated through theprocedures described with reference to FIGS. 5 to 8 may be too small forthe object 871 to fit in, i.e., the bounding box may be inaccurate.Measuring the object in this way with only with one ToF frame maygenerate a bounding box inaccurate in size with respect to the actualobject. Likewise, when measuring the object in an excessively dark orbright place or using part of an image (e.g., ToF frame) of the object,an inaccurate bounding box can be generated.

According to an embodiment, if it is determined that the bounding boxgenerated as described above is inaccurate, the electronic device 800may display guide information prompting the user to move the electronicdevice 800 to a different location to take another image (or depthinformation) of the corresponding object.

The description is made hereinafter of the procedure for displayingguide information with reference to FIGS. 9A to 9D.

FIGS. 9A to 9D are diagrams illustrating exemplary screen displays forexplaining a procedure for an electronic device to display objectmeasurement-related guide information according to a disclosedembodiment.

According to an embodiment, in the case where a first bounding box 990generated as described as above differs from the actual region of theobject, the processor (e.g., processor 340 in FIG. 3) may display guideinformation 935 prompting regeneration of the bounding box in adifferent direction.

In reference to FIG. 9A, the first bounding box 990 overlaid on an imageof the object 971 taken by the RGB image sensor (e.g., RGB image sensor320 in FIG. 3) is smaller in size than the actual region of the object971.

According to an embodiment, the processor may extract a regioncorresponding to the object 971 in the first RGB image taken by the RGBimage sensor, determine that the first bounding box is overlaid on theregion of the object 971 in the first RGB image, and compare the size ofthe overlapping part with a threshold value. If the result of thecomparison shows that the size of the overlapping part is less than thethreshold value, this means that the first bounding box 990 isinaccurate; thus, the processor may display the guide information 935 onthe display 930.

In reference to FIG. 9B, the first bounding box may cover part of theobject 971. This is because the object 971 is partly within the viewingangle of the ToF camera when the first ToF frame was acquired forgenerating first depth information.

According to another embodiment, if it is determined that the regioncorresponding to the object 971 in the RGB image is partly out of theviewing angle of the RGB image sensor, the processor may display guideinformation 936 prompting regeneration of the bounding box in adifferent direction.

Because the ToF sensor and the RGB image sensor are in the electronicdevice, the ToF sensor and the RGB image sensor may have the same viewangle. Accordingly, the processor may determine whether the object iswholly within the viewing angle of the ToF sensor based on the RGB imageacquired by the RGB image sensor. The processor may determine whetherthe object is partly out of the viewing angle based on pixel values atthe vertical and horizontal edges of the RGB image.

In reference to FIG. 9C, a first point cloud 982 generated as describedabove may be composed of a relatively small number of points because theelectronic device 900 is too far from the object 971.

According to an embodiment, if the number of points of the first pointcloud 982 is equal to or less than a threshold, the processor maydisplay guide information 937 on the display 930 prompting regenerationof the bounding box from a shorter distance.

In reference to FIG. 9D, in the case of generating a point cloud 983 ofthe object 971 when the ambient light is too dark or too bright, thepoint cloud 983 may be composed of a small number of points because theToF sensor emits light in the infrared band. In this case, the processormay display guide information 938 prompting use of the RGB image sensorbecause the RGB image sensor would provide a more accurate point cloud.

According to an embodiment, in the case where the number of points ofthe first point cloud generated based on the first ToF frame is equal toor less than the threshold, the processor may display guide informationprompting use of the RGB image sensor for generating the bounding box onthe display. The processor may also check for the ambient brightnessbased on an RGB image frame taken by the RGB image sensor and, if it isdetermined that the ambient brightness is too bright or too dark,display guide information 938, on the display, prompting use of the RGBimage sensor for generating the bounding box.

FIG. 10 is a diagram illustrating an exemplary screen display forexplaining a point cloud of an object that is generated by an electronicdevice according to a disclosed embodiment.

After displaying a first bounding box (e.g., first bounding box 890 inFIG. 8), the processor may acquire second depth information according toa predetermined logic and/or after displaying the guide information asdescribed with reference to FIGS. 9A to 9D. For example, the user maymove the electronic device 1000 to a new location (e.g., secondlocation) to take an image of the object from the new location accordingto the guide information. In the embodiments of FIGS. 10 to 12, it isassumed that the electronic device is horizontally moved around theobject 1071 to measure the object 1071 in a second direction forming apredetermined angle with a first direction. However, the movingdirection is not limited to the disclosed embodiments, and the seconddirection may be a direction tilted upward or downward from the firstdirection or in the opposite direction of the first direction.

According to an embodiment, the processor may generate the second depthinformation in the second direction from the object.

The processor may generate a second point cloud 1080 based on the seconddepth information. The second point cloud 1080 may be generated by usingthe same procedure as that for the first point cloud.

The processor may display the second point cloud 1080 such that pointsof the second point cloud 1080 are overlaid on corresponding objects1071 and 1072.

FIG. 11 is a diagram illustrating an exemplary screen display forexplaining a procedure for an electronic device to generate two boundingboxes for an object according to a disclosed embodiment.

A processor (e.g., processor 340 in FIG. 3) may remove a referencesurface from a second point cloud (e.g., second point cloud 1080 in FIG.10) and generate a second bounding box 1192. The second bounding box1192 may be generated in the same manner as that used for the firstbounding box 1191.

In reference to FIG. 11, the electronic device may display the first andsecond bounding boxes 1191 and 1192 on a display 1130. The first andsecond bounding boxes 1191 and 1192 may each have an overlapping partoverlapping the object and a non-overlapping part that does not overlapthe object.

FIG. 12 is a diagram illustrating an exemplary screen display forexplaining a procedure for an electronic device to update a bounding boxaccording to a disclosed embodiment.

According to an embodiment, a processor (e.g., processor 340 in FIG. 3)may generate a third bounding box 1293 by combining a first bounding box(e.g., first bounding box 1191 in FIG. 11) and a second bounding box(e.g., second bounding box 1192 in FIG. 11).

The third bounding box 1293 may be in the shape of a cuboid having asize sufficient to fit the first and second bounding boxes. That is, thethird bounding box 1293 may be generated based on the smallest andlargest of the x-, y-, and z-axis coordinate values of the first andsecond bounding boxes.

The processor may display the third bounding box on a display 1230 suchthat the third bounding box is overlaid on an object 1271.

According to an embodiment, the processor may display the third boundingbox along with numeric information 1299 of the third bounding box 1293.Here, the numeric information 1299 may include edge lengths and thevolume of the third bounding box. The processor may generate the numericinformation 1299 based on the smallest and largest coordinate values ofpoints of a point cloud corresponding to the third bounding box 1293.

FIGS. 13A and 13B are diagrams illustrating exemplary screen displaysfor explaining a procedure for an electronic device to generate abounding box of a 2-dimensional object according to a disclosedembodiment.

According to an embodiment, the electronic device 1300 (e.g., electronicdevice 300 in FIG. 3) may generate a bounding box for a 2-dimensionalobject (e.g., rectangle and ellipse) on a plane.

In reference to FIG. 13A, a paper document 1371 may be placed on a table1372. If no 3-dimensional objects are detected after attemptingmeasurement of 3-D objects using the ToF sensor (e.g., ToF sensor 310 inFIG. 3) and/or the RGB image sensor (e.g., RGB sensor 320 in FIG. 3) orif the user selects a 2-dimensional object measurement option, aprocessor (e.g., processor 340 in FIG. 3) may perform a process forgenerating a bounding box for a 2-dimensional object.

The processor may generate a point cloud as denoted by reference numbers1381 and 1382 based on depth information acquired by means of the ToFsensor and/or the RGB sensor and display the point cloud 1381 and 1382on a display 1330. The point cloud 1381 and 1382 may include all pointscorresponding to the paper document 1371 and the table 1372.

The processor may generate a point cloud mask with the points croppedamong the generated point cloud that correspond to the paper document1371 as a 2-dimensional object. For example, the processor may detect anarea with a predetermined shape using various edge extraction methodsand may generate the point cloud mask by cropping the area.

The processor 340 may generate a 2-dimensional bounding box (e.g.,rectangle) based on the smallest and largest x- and y-axis coordinatevalues of the points of the point cloud mask.

In reference to FIG. 13B, the processor may display the generatedbounding box such that the bounding box is overlaid on the image of thepaper document 1371 as a 2-dimensional object. According to anembodiment, the processor may display numeric information 1399 includingat least one of the side length and/or area of the bounding box of thepaper document 1371 as the 2-dimensional object.

According to an embodiment of FIGS. 13A and 13B, the electronic device1300 is capable of automatically calculating the size of the2-dimensional object without any intervention of the user for setting upreference points for measurement, unlike conventional technologies. Theproposed method is capable of calculating the size of the bounding boxaccurately even when some depth information is missing.

According to an embodiment, an electronic device 300 may include adisplay 330, a memory 350, and a processor 340 that is operativelyconnected to the display 330 and the memory 350. The memory 350 maystore instructions that, when executed by the processor, cause theprocessor to generate first depth information in a first direction froman object, generate a first point cloud of the object based on the firstdepth information, generate a first bounding box containing one or morepoints of the first point cloud, generate second depth information in asecond direction from the object, the second direction being differentfrom the first direction, generate a second point cloud of the objectbased on the second depth information, generate a second bounding boxcontaining one or more points of the second point cloud, generate athird bounding box by combining the first and second bounding boxes, anddisplay the third bounding box on the display 330.

According to an embodiment, the instructions may further cause theprocessor to display numeric information including a volume of the thirdbounding box and a length of an edge of the third bounding box on thedisplay 330.

According to an embodiment, the instructions may further cause theprocessor to generate the numeric information based on the smallest andlargest coordinate values of points in the third bounding box.

According to an embodiment, the electronic device 300 may furtherinclude an RGB image sensor 320, and the instructions may further causethe processor to display the third bounding box on an RGB imageincluding the object that is taken by the RGB image sensor 320.

According to an embodiment, the instructions may further cause theprocessor to extract the object from a first RGB image taken by the RGBimage sensor 320 in the first direction from the object, identify anoverlapping region where the first bounding box is overlapped with aregion of the object extracted from the first RGB image, and displayguide information on the display 330 prompting acquisition of the seconddepth information in the second direction based on a size of theoverlapping region being equal to or less than a threshold.

According to an embodiment, the instructions may further cause theprocessor to display guide information on the display 330 promptingacquisition of the second depth information in the second direction, thedisplay of the guide information may be based on at least part of theobject being out of a viewing angle of an RGB image sensor.

According to an embodiment, the instructions may further cause theprocessor to display guide information on the display 330 promptingacquisition of the second depth information at a location closer to theobject based on a number of points in the first point cloud being equalto or less than a threshold.

According to an embodiment, the instructions may further cause theprocessor to detect a reference surface in the first point cloudgenerated based on the first depth information, where the first pointcloud includes points above the reference surface.

According to an embodiment, the instructions may further cause theprocessor to generate multiple first point clouds corresponding tomultiple objects separated from each other by removing points in thefirst point cloud corresponding to the reference surface and generatefirst bounding boxes corresponding to the multiple objects.

According to an embodiment, the electronic device 300 may furtherinclude a time of flight (ToF) sensor 310, and the instructions mayfurther cause the processor to generate the first depth informationbased on a first ToF frame taken by the ToF sensor 310 in the firstdirection from the object and generate the second depth informationbased on a second ToF frame taken by the ToF sensor 310 in the seconddirection from the object.

According to an embodiment, the electronic device 300 may furtherinclude an RGB image sensor 320, and the instructions may further causethe processor to display guide information on the display 330 promptingacquisition of the second depth information using the RGB image sensor320 based on a number of points in the first point cloud acquired fromthe first ToF frame being equal to or less than a threshold.

According to an embodiment, the electronic device 300 may furtherinclude an RGB image sensor 320, and the instructions may further causethe processor to generate the first depth information based on a firstdifference between a third RGB frame taken by the RGB image sensor 320in a third direction from the object and a first RGB frame taken by theRGB image sensor 320 in the first direction from the object and generatethe second depth information based a second difference between the firstRGB frame and a second RGB frame taken by the RGB image sensor 320 inthe second direction from the object.

FIG. 14 is a flowchart illustrating an object measurement method of anelectronic device according to a disclosed embodiment.

In the embodiment of FIG. 14, the object measurement method may beperformed by an electronic device (e.g., electronic device 300 in FIG.3) described with reference to FIGS. 1 to 12, and the technical featuresdescribed above are omitted below.

At operation 1410, the electronic device (e.g., electronic device 300 inFIG. 3) may generate first depth information in a first direction froman object (e.g., object 220 in FIG. 2). According to an embodiment, theelectronic device may acquire the first depth information of the objectby means of a ToF sensor (e.g., ToF sensor 310 in FIG. 3) or an RGBimage sensor (e.g., RGB sensor 320 in FIG. 3).

At operation 1420, the electronic device may generate a first pointcloud of the object based on the first depth information. According toan embodiment, the electronic device may generate the first point cloudcomposed of only the points above a reference surface among the pointcloud generated based on the first depth information.

At operation 1430, the electronic device may generate a first boundingbox containing the first point cloud. The electronic device may generatethe first bounding box using the smallest and largest x-, y-, and z-axiscoordinate values of the points obtained from the coordinate informationof the points constituting the first point cloud.

The electronic device may display the first bounding box such that thefirst bounding box is overlaid on the object. If it is determined thatthe first bounding box is inaccurate, the electronic device may displayguide information, on a display, prompting generation of a secondbounding box in a different direction from the object.

According to an embodiment, if it is determined that the first boundingbox differs in part from a region including the actual object, theprocessor may identify that the first bounding box is inaccurate anddisplay guide information prompting generation of the second boundingbox in a different direction.

According to an alternative embodiment, if the object extracted from afirst RGB image is partly out of the viewing angle of the RGB imagesensor, the processor may display guide information prompting a shift inthe location of the electronic device.

According to an alternative embodiment, if the number of pointsconstituting the first point cloud is less than a threshold, theprocessor may display guide information prompting a shift in thelocation of the electronic device to be closer to the object.

According to an alternative embodiment, if the number of pointsconstituting the first point cloud acquired from a first ToF frame takenby the ToF sensor 310 is less than a threshold, the processor maydisplay guide information prompting acquisition of depth informationwith the RGB image sensor.

How to display the guide information has been described with referenceto FIGS. 9A to 9D.

At operation 1440, the electronic device may generate second depthinformation in a second direction from the object. The second depthinformation may be generated in the same manner as that of the firstdepth information as described with reference to operation 1410.

At operation 1450, the electronic device may generate a second pointcloud of the object based on the second depth information. The secondpoint cloud may be generated in the same manner as that of the firstpoint cloud as described with reference to operation 1420.

At operation 1460, the electronic device may generate a second boundingbox containing the second point cloud. The second bounding box may begenerated in the same manner as that of the first bounding box asdescribed with reference to operation 1430.

At operation 1470, the electronic device may generate a third boundingbox by combining the first and second bounding boxes.

At operation 1480, the electronic device may display the third boundingbox. The electronic device may also display numeric information (e.g.,edge lengths and volume) of the third bounding box.

According to an embodiment, an object measurement method of anelectronic device may include generating first depth information in afirst direction from an object, generating a first point cloud of theobject based on the first depth information, generating a first boundingbox containing one or more points of the first point cloud, generatingsecond depth information in a second direction from the object, thesecond direction differing from the first direction, generating a secondpoint cloud of the object based on the second depth information,generating a second bounding box containing one or more points of thesecond point cloud, generating a third bounding box by combining thefirst and second bounding boxes, and displaying the third bounding box.

According to an embodiment, the method may further include displayingnumeric information including a volume of the third bounding box and alength of an edge of the third bounding box.

According to an embodiment, the numeric information may be generatedbased on the smallest and largest coordinate values of points in thethird bounding box.

According to an embodiment, the method may further include extractingthe object from a first red, green, and blue (RGB) image taken by an RGBimage sensor 320 in the first direction from the object, identifying anoverlapping region where the first bounding box is overlapped with aregion of the object extracted from the first RGB image, and displayguide information prompting acquisition of the second depth informationin the second direction based on a size of the overlapping region beingequal to or less than a threshold.

According to an embodiment, the method may further include displayingguide information prompting acquisition of the second depth informationin the second direction on a display 330, where the displaying of theguide information may be based on at least part of the object being outof a viewing angle of the RGB image sensor 320.

According to an embodiment, the method may further include display guideinformation prompting to acquire the second depth information at alocation closer to the object based on a number of points in the firstpoint cloud being equal to or less than a threshold.

According to an embodiment, the generating of the first point cloud ofthe object may further include detecting a reference surface in thefirst point cloud generated based on the first depth information, wherethe first point cloud may include points above the reference surface.

According to an embodiment, the first depth information may be generatedusing a time of flight (ToF) sensor 310.

As described above, the electronic device and object measurement methodthereof disclosed in the certain embodiments is advantageous because theelectronic device can automatically and accurately measuring the size ofa real-world object and guiding the user to a more accurate measurementmethod.

Certain of the above-described embodiments of the present disclosure canbe implemented in hardware, firmware or via the execution of software orcomputer code that can be stored in a recording medium such as a CD ROM,a Digital Versatile Disc (DVD), a magnetic tape, a RAM, a floppy disk, ahard disk, or a magneto-optical disk or computer code downloaded over anetwork originally stored on a remote recording medium or anon-transitory machine readable medium and to be stored on a localrecording medium, so that the methods described herein can be renderedvia such software that is stored on the recording medium using a generalpurpose computer, or a special processor or in programmable or dedicatedhardware, such as an ASIC or FPGA. As would be understood in the art,the computer, the processor, microprocessor controller or theprogrammable hardware include memory components, e.g., RAM, ROM, Flash,etc. that may store or receive software or computer code that whenaccessed and executed by the computer, processor or hardware implementthe processing methods described herein.

While the present disclosure has been shown and described with referenceto various embodiments thereof, it will be understood by those skilledin the art that various changes in form and details may be made thereinwithout departing from the present disclosure as defined by the appendedclaims and their equivalents.

What is claimed is:
 1. An electronic device comprising: a display; a memory; and a processor operatively connected to the display and the memory, wherein the memory stores instructions that, when executed by the processor, cause the processor to: generate first depth information in a first direction from an object, generate a first point cloud of the object based on the first depth information, generate a first bounding box containing one or more points of the first point cloud, generate second depth information in a second direction from the object, the second direction being different from the first direction, generate a second point cloud of the object based on the second depth information, generate a second bounding box containing one or more points of the second point cloud, generate a third bounding box by combining the first and second bounding boxes, and display the third bounding box on the display.
 2. The electronic device of claim 1, wherein the instructions further cause the processor to display numeric information including a volume of the third bounding box and at least one length of an edge of the third bounding box on the display.
 3. The electronic device of claim 2, wherein the instructions further cause the processor to generate the numeric information based on smallest and largest coordinate values of points in the third bounding box.
 4. The electronic device of claim 1, further comprising a red, green, and blue (RGB) image sensor, wherein the instructions further cause the processor to display the third bounding box on an RGB image including the object that is taken by the RGB image sensor.
 5. The electronic device of claim 4, wherein the instructions further cause the processor to: extract the object from a first RGB image taken by the RGB image sensor in the first direction from the object, identify an overlapping region where the first bounding box is overlapped with a region of the object extracted from the first RGB image, and display guide information prompting acquisition of the second depth information in the second direction on the display based on a size of the overlapping region being equal to or less than a threshold.
 6. The electronic device of claim 1, wherein the instructions further cause the processor to display guide information prompting acquisition of the second depth information in the second direction on the display, the display of the guide information is based on at least part of the object being out of a viewing angle of a red, green, and blue (RGB) image sensor.
 7. The electronic device of claim 1, wherein the instructions further cause the processor to display guide information prompting acquisition of the second depth information at a location closer to the object on the display based on a number of points in the first point cloud being equal to or less than a threshold.
 8. The electronic device of claim 1, wherein the instructions further cause the processor to detect a reference surface in the first point cloud generated based on the first depth information, and wherein the first point cloud includes points above the reference surface.
 9. The electronic device of claim 8, wherein the instructions further cause the processor to: generate multiple first point clouds corresponding to multiple objects separated from each other by removing points in the first point cloud corresponding to the reference surface, and generate first bounding boxes corresponding to the multiple objects.
 10. The electronic device of claim 1, further comprising a time of flight (ToF) sensor, wherein the instructions further cause the processor to generate the first depth information based on a first ToF frame taken by the ToF sensor in the first direction from the object and generate the second depth information based on a second ToF frame taken by the ToF sensor in the second direction from the object.
 11. The electronic device of claim 10, further comprising a red, green, and blue (RGB) image sensor, wherein the instructions further cause the processor to display guide information on the display, the guide information prompting acquisition of the second depth information using the RGB image sensor based on a number of points in the first point cloud acquired from the first ToF frame being equal to or less than a threshold.
 12. The electronic device of claim 1, further comprising a red, green, and blue (RGB) image sensor, wherein the instructions further cause the processor to: generate the first depth information based on a first difference between a third RGB frame taken by the RGB image sensor in a third direction from the object and a first RGB frame taken by the RGB image sensor in the first direction from the object, and generate the second depth information based a second difference between the first RGB frame and a second RGB frame taken by the RGB image sensor in the second direction from the object.
 13. An object measurement method of an electronic device, the method comprising: generating first depth information in a first direction from an object; generating a first point cloud of the object based on the first depth information; generating a first bounding box containing one or more points of the first point cloud; generating second depth information in a second direction from the object, the second direction being different from the first direction; generating a second point cloud of the object based on the second depth information; generating a second bounding box containing one or more points of the second point cloud; generating a third bounding box by combining the first and second bounding boxes; and displaying the third bounding box.
 14. The method of claim 13, further comprising displaying numeric information including a volume of the third bounding box and at least one length of an edge of the third bounding box.
 15. The method of claim 14, wherein the numeric information is generated based on smallest and largest coordinate values of points in the third bounding box.
 16. The method of claim 13, further comprising: extracting the object from a first red, green, and blue (RGB) image taken by an RGB image sensor in the first direction from the object; identifying an overlapping region where the first bounding box is overlapped with a region of the object extracted from the first RGB image; and displaying guide information prompting acquisition of the second depth information in the second direction based on a size of the overlapping region being equal to or less than a threshold.
 17. The method of claim 13, further comprising displaying guide information prompting acquisition of the second depth information in the second direction on a display, wherein the displaying of the guide information is based on at least part of the object being out of a viewing angle of an RGB image sensor.
 18. The method of claim 13, further comprising displaying guide information prompting acquisition of the second depth information at a location closer to the object based on a number of points in the first point cloud being equal to or less than a threshold.
 19. The method of claim 13, wherein the generating of the first point cloud of the object further comprises: detecting a reference surface in the first point cloud generated based on the first depth information, wherein the first point cloud includes points above the reference surface.
 20. The method of claim 13, wherein the first depth information is generated using a time of flight (ToF) sensor. 